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ABSTRACT 

In this note we discuss techniques for determining the auto- 
morphism group of a genus g hyperelliptic curve X g defined 
over an algebraically closed field k of characteristic zero. 
The first technique uses the classical GZ/2(fc)-invariants of 
binary forms. This is a practical method for curves of small 
genus, but has limitations as the genus increases, due to the 
fact that such invariants are not known for large genus. 

The second approach, which uses dihedral invariants of 
hyperelliptic curves, is a very convenient method and works 
well in all genera. First we define the normal decomposition 
of a hyperelliptic curve with extra automorphisms. Then 
dihedral invariants are defined in terms of the coefficients of 
this normal decomposition. We define such invariants inde- 
pendently of the automorphism group Aut(A" 9 ). However, 
to compute such invariants the curve is required to be in its 
normal form. This requires solving a nonlinear system of 
equations. 

We find conditions in terms of classical invariants of bi- 
nary forms for a curve to have reduced automorphism group 
A4, S4, A$. As far as we are aware, such results have not 
appeared before in the literature. 
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1. INTRODUCTION 

Let X g be an algebraic curve of genus g defined over an 
algebraically closed field k of characteristic zero. We denote 
by Aut(Xg) the group of analytic (equivalently, algebraic) 
automorphisms of X g . Then Aut(X g ) acts on the finite set 
of Weierstrass points of Xg. This action is faithful unless 
X g is hyperelliptic, in which case its kernel is the group of 
order 2 containing the hyperelliptic involution of X g . Thus 
in any case, Aut(A' 9 ) is a finite group. This was first proved 
by Schwartz. In 1893 Hurwitz discovered what is now called 
the Riemann-Hurwitz formula. From this he derived that 

|Aut(* g )| < 84( fl -l) 

which is known as the Hurwitz bound. However, it is not 
an easy task to compute the automorphism group of a given 
algebraic curve. Even compiling a list of possible candidates 
for a small genus g is quite difficult. In |10) we provide an 
algorithm which computes such lists. We give a complete 
list for g — 3 and list "large" groups for g < 10. This work 
is based on previous work of Breuer, among many others; 
see for a complete list of references. 

If X g is hyperelliptic then Aut(X g ) is a degree 2 central 
extension of Z„, D n , A4, £4, A5. We will explain this briefly 
in section 2. However, computing Aut(A" g ) for a given X g is 
still difficult. Even sophisticated computer algebra packages 
do not have such capabilities for g > 3. The case g — 2 has 
recently been implemented in Magma ;fj and is based on 
methods used in |15|. 

In this short note we will focus on determining Aut(A" s ) 
for a given genus g hyperelliptic curve X g . We will not prove 
any of the results. The interested reader can check 0, 
1 121 . |13|. |14|. or |15| for details. Most of the papers above 
have focused on studying the locus of all hyperelliptic curves 
of genus g whose automorphism group contains a subgroup 
G and inclusions between such loci. In this paper we com- 
bine the above results to get a treatment for all hyperelliptic 
curves in all genera. We generalize the notion of dihedral 
invariants of hyperelliptic curves with extra involutions dis- 
covered in |S] for all hyperelliptic curves with extra automor- 
phisms (cf. Theorem 5.1.). Using these dihedral invariants 
and classical invariants of binary forms of degree 2g + 2 we 
discover some nice necessary conditions for a curve to have 
reduced automorphism group A4,Sa,As (cf. section 5). 

Notation: We will use the term "curve" to mean a "com- 
pact Riemann surface" . Throughout this paper X g denotes a 
hyperelliptic curve of genus g > 2. D n denotes the dihedral 
group of order 2n. 



2. PRELIMINARIES 

Let k be an algebraically closed field of characteristic zero 
and X g be a genus g hyperelliptic curve given by the equa- 
tion Y 2 = F(X), where deg(F) = 2g + 2. Denote the func- 
tion field of X g by K := k(X, Y). Then, k(X) is the unique 
degree 2 genus zero subfield of K. K is a quadratic exten- 
sion field of k(X) ramified exactly at d = 2g + 2 places 
a.i,...,a.4 of k(X). The corresponding places of K are 
called the Weierstrass points of K. Let V := {ai, ■ ■ ■ , ctd} 
and G = Aut(K/k). Since k(X) is the only genus sub- 
field of degree 2 of K, then G fixes fc(X). Thus, Go := 
Gal(K/k(X)) — (zo), with zq = 1> is central in G. We call 
the reduced automorphism group of K the group G := G/Gq. 
By a theorem of Dickson, G is isomorphic to one of the fol- 
lowing: 

Z n , Dn, A4, S4, A5, 

with branching indices of the corresponding cover (j> : P 1 — > 
P 1 /G given respectively by 

(n, n), (2, 2, n), (2, 3, 3), (2, 4, 4), (2, 3, 5). (1) 

In all subgroups of G are classified and in JJ all groups 
that occur as full automorphism groups of hyperelliptic curves 
are classified. We use the notation of [3] and define V n , H n , 
G n , U n , W2, W3 as follows: 
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The following is proven in 

Theorem 2.1. The automorphism group of a hyperellip- 
tic curve is one of the following D n , Z, n , V„, H n , G„, U n , 
GL 2 (3), W 2 , W 3 . 

The reader should be careful when reading Theorem 3.1., 
in |3| . It seems as the cases Hi and Gi (which are isomorphic 
to Z4) must be excluded. For example, for g = 2, according 
to Theorem 3.1., Hi = 1,4 must occur as an automorphism 
group, but it is well known that this is not the case; see 1151 
among many others. It is safe to exclude these cases since 
the group is cyclic and corresponds to case 3 of Table 1. 

Also, for g = 3 let N = 3 in the case 3.d, of Table 2 in 
3 . This case is not excluded from Theorem 3.1., (pg. 273). 
In this case the group is D3 (dihedral group of order 6) and 
this group does not occur as an automorphism group of a 
genus 3 hyperelliptic curve; see |10|. 

2.1 MODULI SPACES OF COVERS 

Let <f>o ■ X g — * P 1 be the cover which corresponds to the 
degree 2 extension K/k(X). Then, tjj :— <f> o cf> has mon- 
odromy group G := Aut(A" 9 ). From basic covering theory, 
the group G is embedded in the group S„, where n = degip. 
There is an r-tuple a := (cti, . . . ,ay), where ai G S„ such 
that o\, . . . ,a r generate G and o~\ • • • a r = 1. The signature 
of ip is an r-tuple of conjugacy classes C := (Gi, . . . , C r ) in 
S n such that d is the conjugacy class of a\. We use the 



notation n v to denote the conjugacy class of permutations 
which are a product of p cycles of length n. Using the signa- 
ture of <j) : P 1 — > P 1 given in Q and the Riemann-Hurwitz 
formula, one finds out the signature of t/j '■ Xg — > P 1 for any 
given g and G. A natural question is if a given group G 
occurs as an automorphism group of a curve X g with more 
then one signature C (cf. Theorem 12. 211 . 

For a fixed G, C the family of covers ifi : X g — > P 1 is a 
Hurwitz space H(G,C). This is a quasiprojective variety, 
not a priori connected. To show irreducibility of Ti.(G, C) 
one has to show that there is only one braid orbit in the set 
of Nielsen classes Ni(G, C). 

There is a map 

$ : H(G, C) -> n g 

where tig is the moduli space of genus g hyperelliptic curves. 
We denote by 8(G, C) the dimension in H g of ®(H(G, C)). 
Further i(G) denotes the number of involutions of G. 

Theorem 2.2. For each g > 2, the groups G that occur 
as automorphism groups and their signatures C are given 
in Table 1. Moreover; TL{G,C) is an irreducible algebraic 
variety of dimension 5{G,C) as given in Table 1. 

Finding algebraic descriptions of Hurwitz spaces is in gen- 
eral a difficult problem. In |14| it is shown that each of the 
spaces Ti[G, C) is a rational variety. Further, the inclusions 
between such loci are studied. 

Let t be the order of an automorphism of an algebraic 
curve X g (not necessary hyperelliptic). Hurwitz [Sj showed 
that t < 10(g — 1). In 1895, Wiman improved this bound 
to be t < 2(2g + 1) and showed that it is the best possible. 
Thus, if a cyclic group H occurs as an automorphism group 
then \H\ < 2(2g + 1). Indeed, this bound can be achieved 
for any genus via a hyperelliptic curve. For example, the 
curve 

Y 2 = X(X 23+1 - 1) 

has automorphism group the cyclic group of order <fg + 2. 
This is the second case in Table 1, when n = 2g + 1. The 
family of such curves is O-dimensional in 7i g . 

Now we turn our attention to determining if a given curve 
Xg belongs to any of the families of Table 1. In other words, 
find conditions in terms of the coefficients of X g such that 
Xg belong to a family in Table 1. This would determine the 
Aut(;f 9 ). 

3. INVARIANTS OF BINARY FORMS 

In this section we define the action of GZ/2(fc) on binary 
forms and discuss the basic notions of their invariants. Let 
k[X, Z] be the polynomial ring in two variables and let Vd 
denote the (d+1) -dimensional subspace of k[X, Z] consisting 
of homogeneous polynomials. 

f{X, Z) = a X d + aiX^Z + ... + a d Z d (3) 

of degree d. Elements in Vd are called binary forms of de- 
gree d. We let Gt/2(fc) act as a group of automorphisms on 
k[X, Z] as follows: 

M=(« ^eGL 2i k),then M (j) = (^) . 

(4) 

This action of GL2(k) leaves Vd invariant and acts irre- 
ducibly on Vd. Let Aq, Ai, ... , A d be coordinate functions 
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Table 1: Automorphism groups of hyperelliptic curves 



on Vd- Then the coordinate ring of Vd can be identified with 
k[A , ...,A d ]. For I G k[Ao, -,Ad] and M G GL 2 {k), define 
I M G k[A , Ad] as follows 



I M (/) := J(M(/)) 



(5) 



for all / G V d . Then J MAr = (J^)^ and Eq. © defines an 
action of GI/2(fc) on A;[j4o, Ad]- A homogeneous polyno- 
mial I G fc[^4o, . . . , Ad, X, Z] is called a covariant of index s 
if 



where 8 — det(M). The homogeneous degree in Ax, ■ ■ ■ , A n 
is called the degree of /, and the homogeneous degree in X, Z 
is called the order of /. A covariant of order zero is called 
invariant. An invariant is a S'L 2 (fc)-invariant on Vd- 

We will use the symbolic method of classical theory to 



construct covariants of binary forms. Let 

f(X,Z) a t X^Z\ 

i=0 ^ ' 

m , v 

g(X,Z) ■.= Y J [^)b i X n - i Z i 



(6) 



be binary forms of degree n and m respectively with coeffi- 
cients in k. We define the r-transvection 



(f,gY := Cfc -^(-l) fc 



fe / r 



fc/ 9A'- fc dY k dX k dY r ~ k 



(7) 



where Ck = — -. It is a homogeneous polynomial 



(m — r) ! (n — r) ! 
n! m! 

in fc[A, Z] and therefore a covariant of order m + 77, — 2r and 
degree 2. In general, the r-transvection of two covariants of 
order m, n (resp., degree p, q) is a covariant of order m + n — 



2r (resp., degree p + q). 

For the rest of this paper F(X, Z) denotes a binary form 
of order d := 2g + 2 as below 



F(X, Z)=Y^ a t X l Z d - 1 = X) ( • ) b * rZ 



rl rvTL — % 



(8) 



where hi — ^"~!/ ! %l -ai, for i = 0, . . . ,d. We denote invariants 
(resp., covariants) of binary forms by I s (resp., J s ) where the 
subscript s denotes the degree (resp., the order). We define 
the following covariants and invariants: 

h ■= (F, F) d , 

J 4] := {F,F) d - 2 \ j = l,...,g, 
I4 := (J4, J4) 4 , 
I4, '■= (Js, Js) 8 , 
/ 6 :=((F,J4) 4 ,(F,J 4 ) 4 ) d - 4 , 
I^.= ((F,J 8 ) 8 ,(F,Js) 8 ) d - S , 
h ■= ((F,J 12 ) 12 ,(F,J 12 ) 12 ) d - 12 , 
h := {F,J d ) d , 

M:= ((F,J 4 ) 4 ,(F,J s ) 8 ) d - 10 , 

7i2 := (M,M) 8 

GI/2(fc)-invariants are called absolute invariants. We de- 
fine the following absolute invariants: 





ik . 




ii 


:= jz, 12 

1 2 


■ — ,3 , 23 ■ 
J 2 




h 


/2 
-"6 


n 


~ 72' S l 


:= W 52 








o 2 


' It ' 





73' 



J ' 2 ' 
J 3 



(4) 2 . 
I12 



Vi 



(10) 



4 := 
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, 5 := 



For a given curve X g we denote by I(X g ) or i(X g ) the cor- 
responding invariants. Two isomorphic hyperelliptic curves 
have the same absolute invariants. 

Remark 3.1. It is an open problem to determine the field 
of invariants of binary form of degree d > 7. 

4. EQUATIONS OF CURVES 

In this section we state the equations of curves in each case 
of Table 1. For a more detailed treatment of these spaces, 
including proofs, the reader can check results in |14| . 
The reader can also check J7J where equations for each family 
are computed; however the main goal of the book is to study 
hyperelliptic Riemann surfaces with real structures. In this 
section G denotes a group as in the first column of Table 
1, and Cf the locus of hyperelliptic genus g curves X g such 
that G is embedded in Aut(X g ). 



4.1 &\yt(x g ) is isomorphic to z n 

If Aut(A" 9 ) =Z n then X g belongs to cases 1, 2, 3 in Table 
1. These loci were studied in detail in |13|. The family of 
curves are given below: 



Y l =X nt + h aiX 



n(t — i) 



+ . 



Y 2 =X nt + ■ 



Y 2 =X (X nt + ■ ■ ■ + ai X 



+ a,X n(t - l) + . 

n(t — i) 



.a t ^X n + h 

,a t - 1 X n + l, (11) 



where t is respectively 2g+2 , 29+1 , — . To classify these curves 
(up to isomorphism) we need to find invariants of the GL2 (k)- 
action on fc(oi, . . • , at-i). The following 

Ui := a 1 ^ 1 at + a 1 ^ 1 at-i, for 1 < i < 8 (12) 

are called dihedral invariants for the genus g and the tuple 

u := (ui, • ■ ■ , u 5 ) 

is called the tuple of dihedral invariants. It can be checked 
that u = if and only if a\ = as = 0. In this case re- 
placing ai,as by a2,as~i in the formula above would give 
new invariants. The next theorem shows that the dihedral 
invariants generate k(£.f). 

Theorem 4.1. Cf is a & -dimensional rational variety. 
Moreover, k(C^) = k(ui, . . . , us). 

If n — 2 then G is the Klein 4-group. Then £f = L g where 
C g is the locus of hyperelliptic curves with extra involutions, 
see A nice necessary and sufficient condition is found in 
|12| in terms of the dihedral invariants for a curve to have 
more than three involutions. More precisely, for such curves 
the relation holds: 

T U± — Ug = 0. 



4.2 Aut(Xg) is isomorphic to D n 

The dihedral group is generated by 



such that 



D„ = (<t,t| a r ' 



<j{X)=e n X, t(X) 



1) 
1 

~~X' 



Then a fixes X = 0, 00 and r fixes X = ±1 and permutes 
and 00. We let 

i 

G(X) :=]J(X 2n + \iX n + l) 



Then, 



G(X) =X 2nt + ai X 2nt - n + ■■■+ a t X nt + 



a t -iX 



(n-l)t 



■aiX n + l 



(13) 



where a;, i = 1, . . . t are polynomials in terms of the symmet- 
ric polynomials si, . . . , St of A, (i.e., a\ — s\, a 2 = t+S2, 03 = 

{t - l)si + s 3 , a 4 := ( n/2 ) + (t - 2)s 2 + s 4 , etc.). 



+ . 



.a t _i* n + l) 



Depending on whether 0, ±1, and cxj are Weierstrass points 
we get the equations Y 2 = F(X) where 

F(X)=G(X), (X n — 1) • G(X), 

X ■ G(X), (X 2n - 1) ■ G(X) (14) 

X{X n - 1) ■ G(X), X (X 2n — 1) ■ G(X) 

where n is respectively as in cases 4-9 of Table 1. 

Remark 4.2. Notice that in all cases n is even; see The- 
orem 2.1., in J^j. 

The case Y 2 = G(X) corresponds to the group Z 2 © D n . If 
n — 2, then this is a special case of G = 1* 2 © Z n . Indeed, 

2 9 - 1 u 2 1 -u 3 g +1 =0 



as expected; see I12| for details. If n > 2 then 

u= (ui,...,u g ) = (0,...,0) 
where u = (ui, . . . , u g ) is as defined in 

4.3 Aut(A^ fl ) is isomorphic to A 4 

This case is treated in detail in |13|. Let 
Gi(X) = X 12 - \X W -33X 8 + 2\X 6 -33X 4 - X t X 2 + 1, (15) 
for A 2 + 108 / 0. Denote by 

s 

G{X) :=l[G t (X) 
i=i 

Then, each family is parameterized as in Table 2. The fol- 
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X(X 4 - 1) • G(X) 
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Table 2: Hyperelliptic curves with Aut(A" 9 ) = A4 

lowing lemma gives a necessary condition that a curve has 
automorphism group Z 2 © A4 or SL2(3). 

Lemma 4.3. Let X g be a hyperelliptic curve of genus g 
with Aut(Xg) = A4. Then, 1 4 = 0. Moreover; 

i) 9 — 4 then I2 = Ia = Ia = Ig = 

ii) if g = 5, 9, 12 then I4 — I§ = 

Hi) if g = 7, 10 t/ien 72 = Ia = I4 = Ig = 
iwj if g — 8 then I4 = 0. 

4.4 Aat(Xg) is isomorphic to s 4 

In this case the reduced automorphism group is generated 
by 

«t(X) = -^, r(X)=iX 
x + 1 

We also denote 

:=X 24 + AX 20 + (759 - 4A)X 16 + 2(3A+ 
1288)X 12 + (759 - 4A)X 8 + XX 4 + 1, 

R(X) :=X 12 - 33X 8 - 33X 4 + 1, ' 
S(X) :=X S + UX 4 + 1, 
T(X) :=X 4 - 1. 

Let 

G(X) ^Hg^X) 
i=i 

where 5 is as in Table 1. Then, the equations of the curves in 
each case are Y 2 = F(X) where F is as below (we suppress 
X): 

F = G, SG, TG, STG, RG, RSG, RTG, RSTG. 



Similar conditions in terms of the classical invariants as in 
the previous case can be obtained in this case also. 

Lemma 4.4. Let X g be a hyperelliptic curve of genus g 
with ~Aui(X g ) = S4. Then, I 4 = 0. 

4.5 Aut{x g ) is isomorphic to A 5 

We briefly state the equations here. We denote by Gi(X), 
R{X), S(X), T(X) the following: 

Gi(X) : = (Xi - 1)X 60 - 36 (I9\i + 29)X 55 + 6(26239Aj - 42079)X 50 

- 540(23199A i - 19343)X 45 + 105(737719Aj - 953143)X 40 

- 72(1815127^^ - 145087)X 35 - 4(8302981A i 4- 49913771)X 30 
+ 72(1815127A i - 145087)X 25 + 105(737719A i - 953143)X 20 
4- 540(23199A i - 19343)X 15 + 6(26239A i - 42079)X 10 

+ 36 (19 Xi + 29)X 5 + (Aj - 1) 

R(X) :=X 30 + 522X 25 - 10005X 20 - 10005X 15 - 522X 5 + 1 
5(X) :=X 20 - 228X 15 + 494X 10 + 228X 5 + 1 
T(X) :=X 10 + 10X - 1. 

(17) 

As above we let 

G(X) :=Y[Gi{X). 

In the order of Table 1 equations are given as Y 2 = F(X) 
where F(X) is as given as (we suppress X): 

F = G, SG, TG, STG, RG, RSG, RTG, RSTG 

These curves can be expressed as Y 2 = M(X 2 ) or Y 2 = 
X ■ M(X 2 ) where M is a polynomial in X 2 . This fact will 
be used in the next section. The expressions are rather 
large and we will not state them here. However, we get the 
following useful fact: 

Lemma 4.5. Let X g be a hyperelliptic curve of genus g 
with ~Aui(X a ) = A 5 . Then, Ia = I' A = h = I'g = I12 = 0. 

5. DETERMINING THE AUTOMORPHISM 
GROUP OF A GIVEN CURVE 

Let X g be given. We want to determine Aut(A" 9 ). In order 
to find an algorithm which would work for any g we would 
have to check whether X g can be written in any of the forms 
above. Thus, we want to find if there is a coordinate change 

^ aX + b 
~* cX + d 

which transforms X g to one of the forms of section 4. This 
would require solving a system of equations for each case 
and therefore would not be efficient. 

5.1 Using classical invariants 

For a fixed g we know the dimension S of the locus £f. We 
compute enough absolute invariants to generate this locus. 
Thus, we determine the loci JZ^ for all G in Table 1 in terms 
of some invariants ii, . . . is+i- These loci are computed only 
once for each g. Then, for a particular curve we simply 
compute these invariants and check if they generate any of 
the loci . These spaces were computed in detail in |14| for 
G = Aa- We will illustrate with g < 12 and G = A 4 , S4, A 5 . 



We define p(X g ) as follows: 



p(X g ) := (pi,p 2 ) = < 
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From Lemma 4.3. and results for cases (5 = 5*4, -As one can 
check that p(<"f s ) is well defined. Moreover, the subvariety 
£.f is 1-dimensional if G is isomorphic to A4, S4, A 5 . For 
each parametric curve X g of the previous section we compute 
p(X g ) in terms of the parameter A. Eliminating A gives an 
equation for £f, see |14) for explicit equations. 

The following algorithm determines if the automorphism 
group of a hyperelliptic genus g < 12 curve is isomorphic to 
Z 2 Aa, I2 © S 4 , Z 2 As, SL 2 (S), SL 2 (5), GL 2 (3), w 3 , w 3 . 

Algorithm 1: 

Input: A hyperelliptic curve X g : Y 2 = F(X,Z). 

Output: Determine if the automorphism group Aut(A" g ) 
is one of Z 2 ©A 4 , Z 2 ®5 4 , Z 2 ©A 5 , SL 2 {3), SL 2 (5), GL 2 (3), 
Wa, W 3 . 

Stepl: Compute l4(X g ). If I4 7^ then Aut(X g ) is not 
isomorphic to any of Z 2 0A 4 , Z 2 ©S 4 , Z 2 ©A 5 , SL 2 (3), SL 2 (5), 
GL 2 (3), W 2 , W 3 . Otherwise go to Step 2. 

Step 2: Compute p{X g ). 

Step 3: Find which is satisfied by p(X g ) (equations 
are given in |13|). Then, Aut(X g ) is isomorphic to G. 

The definition of p(X g ) is a little more elaborate for G = 
Z n , D n since the dimension of is > 1. Once the definition 
of the moduli point is modified and the corresponding Cf 
are computed the following can be used: 

Algorithm 2: 

Input: A hyperelliptic curve X g : Y 2 = F(X, Z). 
Output: The automorphism group Aut(A" g ). 

Step 1: Compute p(X g ). 

Step 2: Find Cf which is satisfied by p(X g ). Then, 
Aut(X g ) is isomorphic to G. 

The above method of classical invariants is difficult to im- 
plement for large g. That's because finding enough absolute 
invariants is not an easy task for large g. Also the expres- 
sions of these invariants and the equations for the loci £f 
get very large as g grows. In order to deal with these prob- 
lems we use the dihedral invariants which will be explained 
next. 

5.2 Using dihedral invariants 

In section 4.1., we introduced dihedral invariants for hy- 
perelliptic curves X g such that Aut(A'g) = Z n . In this section 
we generalize this approach to all hyperelliptic curves with 
extra automorphisms. Theorem 5.1., makes this generaliza- 
tion possible. 

Let X g be an hyperelliptic curve with extra automor- 
phisms. The following lemma gives a general description 
of how to write an equation for X g . 



Theorem 5.1. Let X g be a hyperelliptic curve with 

1^(^)1 > 2. 

Then, X g can be written as 

Y 2 = F(X n ), or Y 2 = X ■ F(X n ), (18) 

where n = 2 or n is odd and divides 2g + 2, 2g + 1, g. More- 
over, if n > 2 then Aut(X g ) is a cyclic group. 

Let X g be a hyperelliptic curve with |Aut(A" 9 )| > 2 and 
written as in 1181 . We call this form a decomposition 
of Xg. Let s be the smallest n that such decomposition is 
possible. Then, 

Y 2 = F(X S ), or- Y 2 — X ■ F(X S ) (19) 

is called the normal decomposition or the normal form 

of X g and s is called the degree of the decomposition. If 
no such decomposition is possible then we say that s — 1. 
Let Xg be in its normal decomposition given below: 

Y 2 =X nt + ■■■ + a l X" it -' ) + ... a t -iX n + 1, 

Y 2 =X {X nt + ■■■ + a l X n(t ~ l) + ... a t _iA n + 1) 

where nt — 2g + 2, 2g + 1, 2g. 
We define the following 

Ui := a\~ l Oj + al~ % a t -i, for l<i<5 = t — 1, (21) 

which are called dihedral invariants for genus g and the tuple 

il 1 := (iti, ... , us) 

is called the tuple of dihedral invariants. It can be checked 
that u = if and only if ai = ag = 0. Then, let (aj, as-j+i) 
be the first nonzero tuple. Replacing a\,ag by aj,as-j+i 
in the formula above would give new invariants. Thus, we 
define 

u\ ■— aj ai + a 5 _ j ag-i+i, (22) 
for 1 < i < 5, and 1 < 3 < [^±1]. Then 

if :=(ui,...,ui l ) (23) 

where m = 8 — 2j. 

Algorithm 3: 

Input: A hyperelliptic curve Xg : Y 2 = F(X, Z). 
Output: The automorphism group Aut(A" 9 ). 

Step 1: Check whether the curve has a normal decompo- 
sition. If "Yes" then go to Step 2 otherwise Aut(A" 9 ) = Z 2 

Step 2: Compute the degree s of the normal decomposi- 
tion. If s is odd then Aut(A" g ) =Z 2s , otherwise go to Step 
3. 

Step 3: Compute the dihedral invariants id\ of the normal 
decomposition. Go to Step 4. 

Step 4: Find £.f which is satisfied by Then, Aut(X g ) 
is isomorphic to G. 

The above method was used in |15| and |Hj to determine 
the automorphism group of genus 2 and 3. It has the advan- 
tages that it can be used for any g no matter how large. A 
disadvantage is that a nonlinear system of equations must 
be solved in order to determine the normal decomposition. 



Example 5.2. For genus 2, the curve can be written as 
Y 2 = X 6 + ai X A + a 2 X 2 + 1 
and its the dihedral invariants are 

ui = a\ + o|, 1*2 = 2a\a2, 

Then, 

a) G = Ve if and only if 112) = (0,0) or 

(ui,u 2 ) = (6750,450). 

b) G^GL 2 {3) if and only if (ui,u 2 ) = (-250,50). 

c) G = Dq if and only if 

u\ - 220u 2 - 16«i + 4500 = 0, 

for u 2 / 18, 140 + 60^5, 50. 

d) G = D4 if and only if 

2u\ - ul = 0, 

for u 2 7^ 2,18,0,50,450. Cases u 2 = 0,450,50 are reduced 
to cases a), and b) respectively, see 1151 for details. 

Remark 5.3. The notation used in ]15)j to denote the 
groups is different. V@ is this case has order 24 and in ]13$ 
is identified as Z3X1D4. 

6. CLOSING REMARKS 

We briefly described techniques of determining the auto- 
morphism group of a hyperelliptic curve. A combination of 
both methods sometime produces better results. Our goal 
is to combine these methods and explicitly compute loci Cf 
for reasonable g (i.e., g < 60). 

There are polynomial time algorithms to compute the de- 
composition of a polynomial F(X) up to an affine transfor- 
mation X — > aX + b, see pj. However, this is not sufficient 
for our purposes since we want to find such decomposition 
up to a liner fractional transformation X — > ^xt-d - ^ a 
polynomial time algorithm would be found in this case this 
would make the second method preferable to the first. 

Besides computing the automorphism groups the above 
techniques can also be used to answer other questions on 
hyperelliptic curves. For example dihedral invariants can 
be used to determine the field of moduli of a given curve. 
The reader can check |12) for details and open questions 
on the field of moduli and other computational aspects of 
hyperelliptic curves. 
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